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ABSTRACT 

The purposes of this study were (1) to develop a 
i.equester-to- system Computer Assisted Instruction (CAI) interface 
with a batch processing information retrieval system, (2) to compare 
users' satisfaction with retrieval services between the experimental 
CAI training approach and the traditional person-to-person 
negotiation process, and (3) to test the ability of the experimental 
CAI interface to impart a knowledge gain of literature-searching 
skills related to the Educational Resources Information Center (ERIC) 
information base. Sixty subjects were randomly assigned to 2 
treatment groups: 30 obtained computex search services by the 
traditional method and 30 were placed directly in charge of 
formulating their own search strategies via the CAI training package. 
Data from 58 subjects were suitable for analysis. The satisfaction 
scores and search precision ratios analyzed indicated that the 
treatment groups did not differ. However, novice users of the CAI 
package acquired significant gains in knowledge relative to the use 
of the ERIC data base. It was concluded that the use of a specially 
created CAI telecommunications interface system is a viable 
alternative to search negotiations which require the services of a 
professional staff. (Author/CH) 
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Purposes and Hypotheses of the Study 

The purposes of the study were to: (a) develop a viable 

reques ter- to-sys tern Computer Assisted Instruction (CAl!) Interface 
with a batch processing automated Information retrieval system;, (b) 
to compare users' satisfaction with retrieval services based upon an 
experimental CAI and traditional search negotiation Interface methods; 
and (c) to test the ability of the experimental CAI Interface to 
Impart a knowledge gain of literature-searching skills related to the 
ERIC Information base. Hypotheses tested were: 

(1) There are no significant differences between the measured 
user satisfaction with computer search results of CAI -Interfaced 
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subjects and those undergoing search negotiations with an information 
specialist. 

(2) There are no significant differences between the subjects' 
pre- and post-measures of knowledge of the information retrieval 
system administered during the experimental CAI treatment. 

(3) There is no significant relationship between measured 
knowledge of the ERIC automated information retrieval system 
acquired as a result of CAI training and user satisfaction as 
measured by the user satisfaction instrument. 

(4) There are no significant differences in user's satis- 
faction between practitioners and researchers as determined by the 
user satisfaction instrument. 

(5) There are no significant interactions between types of 
users (practioner or researcher) and treatments (CAI or search 
negotiations) as measured by the user satisfaction instrument. 
Procedures 

Based upon the careful stuciy of successful past search 
negotiation strategies, a totally conversational CAI training and 
literature search submission program named ERIC/QUERY Interface 
Program (EQUIP) was created taking advantage of existing APL (A 
Programming Language) telecommunications capabilities at New Mexico 
State University. Sixty subjects were assigned to two treatment 
groups: 30 obtained computer search services by means of search 
negotiation practices, and 30 were placed directly in charge of 
formulating their own search strategies via EQUIP. Analysis of data 
collected for 58 subjects was used to test the study hypotheses. 

vii 
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Statistical procedures Included: (a) linear regression, (b) Student's 

^ test, (c) analysis of variance, and (d) Pearson product-moment 
correlation. 

Findings 

The satisfaction scores and seflrch precision ratios analyzed 
Indicated that the treatment groups did not differ. It was shown 
that novice users of EQUIP (Interfaced with QUERY, and ERIC batch 
processing Information retrieval program) acquired significant pre- 
and posttq^t gains. Additionally, It was determined that: (a) the 

retrieval search precision of the two treatment groups did not differ, 
and (b) low search precision Influenced total satisfaction scores of 
users of EQUIP. "' 

Conclusions 

The use of a specially created CAI telecommunications .Interface 
between uninitiated users and a complex automated Information 
retrieval system Is a viable alternative to search negotiations which 
require the services of a professional staff. 




v111 



8 



1 



o 

ERIC 



Table of Contents 

Page 

List of Tables xiii 

List of Figures xiv 

Chapter 

1. Introduction 1 

The Problem 2 

The Objectives 2 

The Hypotheses 3 

The Basic Assumptions of the Study 4 

Limitations 4 

Definition of Terms 5 

Organization of the Study 7 

2, Review of Literature 9 

Theoretical Bases 9 

General systems theory . 9 

Information theory 9 

Decision theory 10 

The Use of Information Retrieval Systems 10 

Batch Processing Automated Information Retrieval 
Systems 12 

Interactive (on-line) Information Retrieval Systems . . 13 

The ERIC/QUERY Information Retrieval System 15 

The Educational Resources Information Center (ERIC) 
retrieval system . 15 

Related Studies 17 

IX 



; • 1 



"i 

4 



I 



i 



9 



Chapter Page 

User-to-systems interface within information 

retrieval programs 17 

The CAI environment as a training method 18 

Other studies 18 

3. The ERIC/QUERY Interface Program (EQUIP) 19 

The Program Language of Equip 19 

The Objectives of EQUIP 20 

The Development of EQUIP 21 

A Generalized Description of EQUIP 22 

Block one: system familiarization 23 

Block two: QUERY familiarization 23 

Block three: search strategies 23 

Block four: practice and submission 23 

The Instructional Sequence of EQUIP 28 

Discussion . 28 

4. Procedures 33 

Design of the Study 33 

Setting of the Stud|y 35 

Summary of the Experimental Design 36 

Sampling Design 36 

The study population 36 

Sampling technique 38 

Limits of inferences to the study population 39 

Measures Employed 39 

Pre- and posttesti ng of EQUIP 40 

X 



o 



Chapter Page 

Implications ..... 71 

The use of EQUIP 71 

Utilization of user characteristics 72 

Recommendations 72 

References 74 

Appendix A. EQUIP 78 

Appendix B. Sample Session . 166 

Appendix C. Instruments and Forms 203 

Appendix D. Technical Notes 207 





4 ' 



xii 




r 






List of Tables 



Table " - Page 



1. A Contrast Between DIALOG and QUERY Information 

Retrieval Systems 16 

2. Subblocks and Objectives of Block 1 of EQUIP 24 

3. Subblocks and Objectives of Block 2 of EQUIP 25 

4. Subblocks and Objectives of Block 3 of EQUIP 26 

5. Subblocks and Objectives of Block 4 of EQUIP 27 

6. Frequencies of Study Sample Subjects in Relation to 

Retrieval Expectations • 45 

7. Study Sample by Treatments 47 

8. Numbers and Percentage of Various Categories of Subjects 

Within Treatment Groups 48 

9. Summary of Statistical Computations Based on User 

Satisfaction Scores 51 

10. Summary Table for Experimental and Control Treatment 

Groups Based on Total Satisfaction Scores Obtained 

on the User Sati sfaction Instrument 53 

11. AOV for Testing Hypothe|es One, Four, and Five 55 




xi ii 




i- 







List of Figures 



Figure - Page- 

1. Major Instructional Sequence of EQUIP 29 

2. An Ideal Research Design 33 

3. The Experimental Research Design 34 

4. Basic Desioii of Experiment 37 

5. Summary Calculations of it' Statistic for Item Three of 

the Satisfaction Questionnaire 50 

6. Summary of Calculations Comparing the Pre- and Posttest 

Interface Knowledge Gains of the Experimental Treatment 
Group 56 

7. Summary of Linear Regression Computational Results 

Relating to the Predictive Relationship of Posttest 
Scores and Pre- and Posttest Gains to Experimental 
Subject's Satisfaction Scores 58 

8. Scattergram of Experimental Group Total Satisfaction 

Scores Versus Pre- and Posttest Gains 60 

9. A Contrast and Correlation of Experimental Total 

Satisfaction Scores and Search Precision for Cluster Two 61 

10. A Contrast of Linear Regression Sutnnary Calculations for the 

Cluster 1 and Cluster 2 Subjects Shown on Figure 8 . . . 62 

11. Summary of Statistical Computation for Determining the 

Significance of Experimental Group Precision Means ... 64 



Chapter 1 : Introduction 



Information is being produced by all disciplines at an 
increasing rate each year. In response to this phehdnrehb^^^ 

United States Office of Education (USOE) has committed millions of 
dollars of the nation's resources to the creation of the first 
national education information system, known as "ERIC (Educational 
Resources Information Center) , to assist the decision-maker, 
researcher, and practitioner assess a growing bank of documents for 
the education profession. 

To aid the serious user in his document search and retrieval 
tasks, in 1969 a computerized searching system known as QUERY was 
created and implemented in over a dozen installations across the 
nation. QUERY is a batch processing system which requires a unique 
"search language" as machine-readable input and offers almost 
unlimited document searching capabilities. Because of this distinctive 
characteristic, if one is to properly use the QUERY automated infor- 
mation system he must be able to select the correct search criterion 
prior to submitting his search request to the computer. 

Currently, all QUERY installations accommodate the information 
seeker (user) by negotiating a search request. This requires a 
conference between the user and an information specialist and is 
concluded when an acceptable search strategy (search language) based 
upon the user's expressed needs is recognized by the information 
specialist. This method of interfacing the system (through the 
information specialist) is less than satisfactory for a number of 
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reasons Including: (a) the difficulty the user has in expressing 

his specific search need to another person; (b) the size of the files 
often "overpowers 'L_the uni t1 ated. us_e^^ the_ EJRI C__systeni ; and -( c )- the- 

search process (via QUERY) is not completely explained to the user, 
thus his knowledge of the retrieval capabilities is severely limited. 
The Problem 

Because of existing difficulties experienced by uninitiated 
users of the ERIC automated Information retrieval system, which 
currently depends upon user interface with the system via information 
specialists, new approaches allowing the potential user of ERIC files 
to meet his information needs must be explored. For the system to 
have maximal impact in the education sector, a more efficient 
interface methodology must be created that will give the potential 
user opportunities to receive relevant experience and traiini^g 
allowing direct interface with the information base.^ y 
The Objectives 

The objectives of this study were; 

(1) To develop. Implement, and test a fully documented 
coriiputer-assisted-instructlon (CAI) package to train potential users 
of the computerized ERIC files to Interface with the Information 
retrieval system without the assistance of an Information specialist. 
Effectiveness of the CAI program as a training package will have been 
assessed through the use of pre- and posttesting. 

(2) To design an Instrument to measure the system user's 
satisfaction with the results of computer searches from 

either the information specialist or from personal Interface with 
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the QUERY Information retrieval system. 

(3) To determine any differences between experimental and 

control groups of ERIG- automated -Information retrieval system users “ 

(Including predesignated subgroups of each)— with the experimental 
group undergoing CAI training on the use of the system and the control 
group receiving no such training — through the use of a user 
satisfaction Instrument and appropriate statistical analysis. 

The Hypotheses 

The specific hypotheses tested were: 

(1) There are no significant differences between the measured 
user satisfaction with computer search results of CAI -Interfaced 
subjects and those undergoing search negotiations with an Information 
specialist. 

(2) There are no significant differences between the subjects' 
pre- and post-measures of knowledge of the information retrieval 
system administered during the experimental CAI treatment. 

(3) There is no significant relationship between measured 
knowledge of the ERIC automated Information retrieval system 
acquired as a result of CAI training and user satisfaction as 
measured by the user satisfaction Instrument. 

(4) There are no significant differences In user 
satisfaction between practitioners and researchers as determined by 
the user satisfaction Instrument. 

(5) There are no significant Interactions between types of 
users (practitioner or researcher) and treatments (CAI or search 
negotiations) as measured by the user satisfaction Instrument. 
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The Basic Assumptions of the Study 

The stu4y Is based on the following assumptions: 

(1) Once an educator understands the utlllty-of an Inf ormatl on- 
base especially designed and maintained for his discipline, he w111 
attempt to capitalize upon the respective benefits of such a system. 

(2) Potential user traits and skills will vary extensively; 
reading ability, typing ability, and educational level (I.e. 
educators vs. trainees) will differ. 

(3) The ERIC Information file size will not remain static; 
file size currently Increases over one thousand documents per month 
(file size was held constant for this study). 

(4) For the duration of the study* computer files, search 
programs, and listing formats will be unchanged. 

(5) Measurement Instruments are valid. 

(6) Subjects have had no prior experience with ERIC automated 
retrieval systems. 

Additionally, it should be noted that the QUERY Information retrieval 
system Is assumed to exist In a stable form. The specific version of 
QUERY used was not altered In any manner for the duration of the study. 
In this manner, the Interface system used was the primary dependent 
variable. Accordingly, QUERY was not an object of Interest for this 
study. 

Limitations 

The systems environment of the New Mexico State University 
(NMSU) Computer Center Is not unlike that of other Installations using 
the QUERY Information retrieval software. There are, however, some 
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unique systems capabilities which are not In existence at many ERIC 
Installations. For example, not all ERIC Installations possess APL 
(A Programming Language) capabilities, and of those that do, not 
many have the ability to Initiate a search from the APL subsystem to 
the main operating job stream. It Is this capability which allows 
for a true Interface of the user to the batch processed QUERY system. 

To be more sped vie, this study Is highly dependent on computer 
capabilities — some of which could be Installed at other computer sites 
In the near future. 

Definition of Terms 

APL . "A Programming Language" Is an IBM-supported telecommuni- 
cations language. The version used for this study Is APL/360-05, 
Program Number 5736-XM6, modified for data-to-job-stream capabilities 
by Dr. Thomas H. Puckett, NMSU Computer Center. 

Automatic Information retrieval. The term "automatic Informa- 
tion retrieval" refers to apy viable computer-based Information 
retrieval system having document Isolation and retrieval capabilities 
using Inverted or sequential files. 

Batch processing . In contrast to on-line processing, batch 
processing Is a technique by which the Items to be processed In a 
data processing machine must be collected Into groups prior to 
processing (Williams, 1965, p. 402). 

Boolean . Algebraic operations on, and functions constructed 
from, two-valued variables, interpreted as representing truth-falsity 
or set membership statCis Is a characteristic of Boolean logic (Williams, 
1965, p. 403). 




UUD ' 



r 



6 

CAI . "Computer Assisted Instruction" 1s a technique which 
capitalizes upon the capabilities of the computer to store and 
selectively provide information to a learner through the use of a 
telecommunicator device. 

CUE. Current Index Jb Journals In. Education is a f 1 1 e 
comprised of citations from more than 500 major educational publica- 
tions: journals, quarterlies, annuals, and yearbooks. 

Descri ptor . Any word or key word used to classify a document 
for subsequent retrieval Is designated a descriptor. 

File . A file refers to a structure of information composed of 
one or more records, such that the records are descriptive of 
individual document citations or classes of citations. 

Hit . A hit Is a retrieved document citation (listing) which 
Is considered relevant to an Information need stated by the user. 

Interface . A method or technique of linking working systems 
without deteriorating the systems Involved Is known as an Interface. 

On-line retrieval system . This term refers to an automated 
information retrieval system which allows the user access to specific 
sections of an Information file via random access techniques and 
provides Infinedlate feedback relating to his search; contrasted with 
batch processed information retrieval. 

Practitioner . For the purposes of this study, a seeker of 
information who desires a small number of highly relevant hits as a 
result of his search effort is classified as a practitioner. 

Researcher . A researcher is a seeker of Information who 
desires a large number of relevant and slightly relevant hits, or 
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who expects a search to retrieve a considerable number of document 
citations with emphasis on "in-depth" criteria. 

RIE . Research in Education is a file comprised of- completed 

research and research-related reports in education as input by the 
various ERIC clearinghouses. 

Search language . A specific encoded Boolean search format used 
to access citations with an automated information retrieval system, 
such as QUERY, is called a search language. 

Search negotiation . Search negotiation is a process whereby 
a user communicates his information need to an ERIC information 
specialist who, in turn, encodes the necessary computer input. 

Search precision . Search precision is a percentage ratio of 
the number of hits a search produces to the total number of document 
citations retrieved. 

Search request . A single search encoded in an acceptable 
search language for QUERY which has been submitted to the computer 
job stream is known as a search request. 

Terminal . An IBM 2741 telecommunication device equipped with 
an APL keyboard and type head (font ball) is known as a terminal. 

User . A user is one who seeks information through the use 
of an automated information retrieval system (QUERY). 

Organization of the Study 

Chapter 2 presents a review of available relevant literature 
and related studies which focus on the problem. Chapter 3 provides 
the major characteristic of EQUIP (ERIC/OUERY Interface Program), the 
experimental treatment considered by this study. A description of 



the procedures used is found in Chapter 4; the analysis of data is 
given in Chapter 5; and findings and conclusions are reviewed in 
Chapter 6. Of possible interest to future implementators of EQUIP 
the entire encoded pack.age is provided in the Appendix as well as 
technical notes and a sample of an EQUIP sample session. 



Chapter 2: Review of Literature 



Of the four sections to this review, the first considers the 
theoretical bases of the problem to be studied, the second deals 

•"V# 

briefly with information retrieval, the third concentrates on the 
ERIC/QUERY Information retrieval effort, and the last focuses on 
related studies. 

Theoretical Basei 

General systems theory . General systems theory as described 
by Optner (1960, p. 9) and Stufflebeam (1971, p. 124) Is concerned 
with two major characteristics — input to the system and output from 
the system. When such systems involve computer usage they "... 
are structured, or designed, to operate in nonvariant, highly 
predictable ways [Optner, 1960, p. 4]." It follows that If a highly 
structured system Is not understood by its users, the efficacy of 
such a system will be less than design expectations. 

This study attempted to show that, within the realm of 
Information retrieval systems, the less knowledge the user of the 
system possesses about the system, the greater the chances the system 
will not operate within Its. designed purpose. 

Information theory . Shannon (1959, p. 18) Indicates that a 
general communication system Is comprised of an Information source 
which originates a message (input), a transmitter, a channel and a 
receiver which processes the message, and a destination (output). 
Meadow (1967, p. 6) renames Shannon's Intermediate processing steps 
(transmitter, channel, and receiver) by referring to them as 
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"transducers" having the capability of selection as well as 
processing. This, according to Meadow, allows for message processing 
through time with less regard for knowing tha identity of the 
originator or the destination. Thus, it seems, the merging of the 
computer system and the Information system allows for Input 
(documents) and output (retrieved documentation) within the context 
of both general systems and information systems. 

Decision theory . It is Interesting to note that the various 
decision-making theories of Kepner and Tregoe (1953, p. 73), 

Griffiths (1959, p. 90), Halpin (1966, p. 35), and Meyer (1968 
vol . 1) each rely on the acquisition of information as a prelude to 
completing the decision-making process. Additionally, each process 
Is described as a system or may be thought of In terms of general 
systems theory. Stuff lebeam (1971, p. 38) points out that the 
availability of information prior to selecting alternatives in 
decision-making is assumed. Thus, the importance of information 
within the context of educational decision-making may not be over- 
looked. 

The Use of Information Retrieval Systems 

Lancaster (1968, p.l), who has been associated with the 
Medical Literature Analysis and Retrieval System (MEDLARS) states: 

"An information retrieval system does not inform (i.e., change the 
knowledge of) the user on the subject of his inquiry. It merely 
informs him of the existence (or nonexistence) and whereabouts of 
documents relating to his request." This suggests that such a 
system may consist of the simplest card catalogue or filing procedure 
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to the highly sophisticated computer-based automated systems which 
exist today. Sal ton (1968) describes such an automated system, 
called SMART, which has been installed at Harvard and Cornell 
Universities. Both Williams (1965, chap. 10) and Lancaster (1968, 
chaps. 1-3) describe traditional and modern Information retrieval 
schemes. Noteworthy In these discussions is the large number of 
variations available to the designers of these systems. 

The use of an automated retrieval system is anything but a 
simple matter. Lancaster (1968 , pp. 181-2) suggests that retro- 
spective literature searches may be divided Into those conducted 
without an intermediary by the person having the information need, 
and those delegated by this person to a second individual, usually 
a librarian or Information specialist. Lancaster (1968, pp. 182-5) 
clearly shows that there are different skills and expectations 
required of the requester under these two schemes. Additionally, 
he points out that twenty per cent of the National Library of 
Medicine's Medical Literature Analysis and Retrieval System's 
searches Involving defective interaction (between the user and the 
searcher) were judged to be of the type in which the requester, 
using an Intermediary, was unable to precisely define his need except 
through some browsing in the literature (1968, p. 184). To 
sunmarlze, this means that to be successful the requester must spend 
a considerable amount of effort defining and negotiating his search 
request with the person who will actually encode the search in 
machine-readable form or learn how to use the system himself. The 
economics of the former appear to be less attractive than those of 
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the latter If it is assumed that the same effectiveness (in terms 
of precision of retrieved information) can be achieved. 

Batch Processing Automated Information Retrieval Systems 

In the fourteen years since the first published application 
to key word indexing of precoordinated information files by the 
computer (Janda, 1968, p. 4), hundreds of batch processing informa- 
tion retrieval systems have been created. Janda (1968) describes 
a number of systems receiving attention within the newly emerging 
discipline of information retrieval. These include: ICJIC (Key- 
Word-In-Context), which uses Ceagaard's BIDAP (BI biographic DAta 
Processor) computer program for input processing; TRIAL (Technique 
to Retrieve Information from Abstracts of Literature); and SEARCH 
(Janda, 1968, p. 28), which uses magnetic tapes and a simple search 
scheme. 

Adams, writing in a collection of papers (Rubinoff, 1965, 
pp. 77-85), describes MEDLARS, which has some similarities to the 
ERIC/QUERY system briefly described in the third section of this 
chapter. 

A review of the literature shows that many batch processing 
information retrieval systems were either designed for highly 
specific applications and/or developed around available computer 
hardware and software. 

The apparent advantages of the batch processing systems 

include: 

(1) Systems designers may capitalize on their existing 
computer capabilities without added costs for additional software. 
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(2) Maximal specialization of the system may be facilitated, 
allowing for "tailor-made" systems which meet the designer's needs 

at the least cost. 

(3) "In-house" created computer programs are maintained 
Internally; thus changes to the system may be made with the 
fewest constraints. 

Because of the acvantages listed, It Is unlikely that batch 
processing Information retrieval systems will be discontinued without 
some significant changes In the costs and "state of the art" of the 
computer Industry. 

It Is noteworthy that, of all of the batch processing Infor- 
mation retrieval systems Investigated by the author, none were 
accompanied with a system-to-user Interface. All required the 
Intervention of an Intermediary, as cited by Lancaster (1968, pp. 
182-5). 

Interactive (on-line) Information Retrieval Systems 

Caruso (1969, pp. 7-18) describes eight on-line Interactive 
retrieval packages. Including Lockheed's DIALOG system which will 
be discussed briefly. In their Independent evaluation of the DIALOG 
system, Tlmble and Coombs (1969, pp. 21-31) present their subjects' 
comments In the form of good and bad features. The favorable features 
of DIALOG were: 

The system Is thorough, comprehensive. 

The system Is effective. 

The system Is generally valuable. 

The system Is exhilarating. Interest-creating. 

Having a hardcopy record to refer to Is handy. 




Unfavorable features were: 

Too many combinations of keys are needed to Input one command. 

Having to build combined sets one step at a time, rather than 
using parentheses, and doing It with one complex statement Is 
Inconvenient, and 

There Is a great deal of "paging" required on the CRT, because 
you can only look at nine terms at a time. 

Additionally, two other characteristics of the DIALOG system are 

suggested: speed, and "horizon widening"— the capability of the 

system to expand or limit a search (kin to browsing). 

The Timble and Coombs evaluation Is based on nine case 
histories and Is not a controlled evaluation of the system. Their 
evaluation does suggest that Interactive on-line systems such as 
DIALOG will meet ERIC users' needs. The estimated cost provided by 
Lockheed for a '-'typical" regular DIALOG user Is $35,000 per year 
for twenty hours of searching per week (Lockheed, 1971b), a cost 
which Is relatively high for most educational Institutions. 

A. R. Sarnum (Rublnoff, 1965, pp. 45-61) describes the Air 
Force's Reliability Centered Data Management System (RCDMS) which 
has on-line retrieval and analysis capabilities. Inherent In the 
on-line systems found In the literature Is the capability of 
allowing the user to browse through the Information file while 
using Immediate feedback— In short, to Interact with the file. 

Some on-line systems such as GRINS and EASY ENGLISH (Caruso, 1969, 
p. 10) use English-language statements as a search criteria, while 
others such as the Lockheed Company's DIALOG (1971a, p. 4) use a 
Boolean search strategy simlllar to that of QUERY. Since a user 
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must be Individually trained to use the system, most Interactive 
(on-line) systems provide a users' training manual or, as was the 
case with Caruso's study (1969), provide CAI tutorial training. 

There exist only two nationally used automated information 
systems designed to access the ERIC Information base, QUERY and 
Lockheed's DIALOG. Table 1 presents a contrast of these two systems 
based on Lockheed's (1971a) user manual and Computer Resources 
Corporation's (1970) Installation manual, and ERIC/CRESS experiences. 
The ERIC/QUERY Information Retrieval System 

The Educational Resources Information Center (ERIC) retrieval 
system . The ERIC system was begun In 1965 with the production and 
dissemination of the Catalog of Selected Documents on the Disadvan- 
taged , and since has grown into an international educational Informa- 
tion retrieval system of considerable scope and complexity (Burchinal, 
1970, pp. 58-63). Currently, ERIC files may be searched manually or 
via the computer using an on-line system called DIALOG or batch 
processing programs such as the USOE -sponsored QUERY system. 

ERIC depends upon the collective efforts of 17 clearinghouses 
located In various areas of the country for the acquisition of 
relevant documents Into the ERIC Information base. One such 
clearinghouse, ERIC/CRESS (Clearinghouse on Rural Education and 
Small Schools), Is located on the New Mexico State University campus. 
In 1969, the USOE-sponsored automated Information retrieval program, 
QUERY, was Installed at ERIC/CRESS. For the first time, educators In 
the Southwest had at their disposal a powerful Information retrieval 
tool to assist them In their professional activities. 
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Table 1 

A Contrast Be^meen DIALOG and QUERY Information Retrieval Systems 



Characteristic 


DIALOG 


QUERY 


Search Input 


Structured 


Structured 


Feedback 


Yes 


No 


Thesaurally Related Terms 


Yes 


Yes (Manuals) 


Count of Citations Retrieved 


Yes 


Yes 


Logic 


Boolean 


Boolean 


Output Format 


Citation 


Citation 


User-to-system Interface 


< 

Yes (Instruction 
Manual ) 


No 


Cost (overall, per year) 


High ($21,000) 


Moderate 
(est. $8,000) 


Turnaround Time 


1-5 days (mailed) 


1-3 days 


Type of Processing 


On-line 

(Interactive) 


Batched 


Communication Mode 


Terminal or CRT Card Reader 
(Cathode Ray Tube) (Punched Cards) 


Browsing Allowed 


Yes 


No 


Computer Facilities 


Central 1 zed 


Decentral 1 zed 
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QUERY Is an advanced, generalized, sequential, file-searching 
computer software package, using RIE and CUE magnetic tapes or disk 
files, which Is capable of Isolating and listing ar\y Informational 
subset, depending on the search strategy used (Computer Resources 
Corporation, 1970). In late 1970, David Altus, while a graduate 
assistant at ERIC/CRESS, modified the QUERY search subprogram to 
expand the power of the search language. Mr. Altus' change to QUERY 
allows the use of a versatile TEXT mod1f1er--a function which allows 
any part of a word, phrase, or sentence to be scanned during the 
record-by- record search operation of QUERY. 

Brandhorst and Marra (1970) provide a comprehensive 
description of RIE and CUE magnetic tape files produced by the 
Leasco Company. A brief file description Is found In appendix D. 

The evaluation of the ERIC system as an Information retrieval 
system as described above Is not considered In this study. Rather, 
the use of the system In relation to the current practice of providing 
an Information specialist as an Intermediary versus the alternative 
of training the requester to formulate his own search strategies Is 
pivotal. The computer files (RIE and CUE) and programs (QUERY) are 
Independent variables. 

Related Studies 

User-to-svstems Interface within Information retrieval programs . 
Caruso (1969) found no sigjjificant (.05 level) differences between an 
experimental group of Mastersrlevel library science students trained 
to operate an on-line Information retrieval system using a computer- 
assisted tutorial program and a control group trained by class 
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lectures. This seems to suggest that subjects may be trained using 
a computer terminal just as effectively as more costly lecture 
classes or Individual person-to-person tutoring. Also, Caruso's 
study was done using an on-line retrieval system— not a batch 
processing system such as QUERY— a difference which Is not critical 
In terms of training, but which does leave the Interface method 
question completely unanswered. 

The CAI environment as a training method . Sutter (1967) found 
no achievement differences between groups of subjects working alone 
with a CAI device (terminal) and those working with the device 
assisted by an Individual tutor. Assuming an effective CAI program 
may be created to train requesters to use the ERIC Information 
retrieval system (computerized), Sutter's findings seem to suggest 
that there will be no need to provide Individual tutors to assist 
In the training process. Combining the results of both Caruso and 
Sutter, within the scope of this study, suggests th^t the use of 
existing computer terminals at New Mexico State University to 
explore the problem Is legitimate. 

Other studies . Although the literature abounds with studies, 
projects, and programs which use CAI as a training media, no CAI 
programs or studies are known to exist which specifically focus upon 
the problem of reques ter- to-sys tern Interface. 
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Chapter 3: The ERIC/QUERY Interface Program (EQUIP) 

Available literature dearly shows the existence of numerous 
batch processing Information retrieval systems as well as an 
Increasing number of on-line Interactive systems. The Investment 
of resources for batch processing systems will most likely Insure 
their existence and maintenance for years to come. The on-line 
systems are gaining In popularity despite their cost, due In part 
to the ability to provide systems-to-users Interface, a feature 
totally lacking with the more numerous, less costly batch processing 
systems. This phenomenon led to the present study. IT It Is 
possible to create a satisfactory Interface package to be coupled to 
a relatively complex QUERY batch processing computer package, greater 
efficacy of the system's use should result. 

Since no search Interface package was known to exist, the 
writer created EQUIP (ERIC/QUERY Interface Program), a CAI training 
and search submission package described in this chapter. 

The Program Language of EQUIP 

i 

IBM's popular telecommunication language, APL, available at 
New Mexico State University, was selected as the language to program 
EQUIP because of Its Immediate availability* versatility, and power 
as a CAI medium. The version of APL used Is not standard, as 
supported by IBM, but Instead has a unique feature provided through 
the efforts of Thomas H. Puckett of the NMSU Computer Center. 

Dr. Puckett's modification of APL allows for the submission of the 
searches encoded by the user directly to the job stream of the 
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university's IBM 360/65 multiprocessing computer job stream— a 
feature not currently available In the IBM version of the language. 

In addition, APL has been found to be efficient as related 
to computer time used for Interface purposes. This characteristic 
coupled with some unique matrix operations and generic time- 
monitoring functions led to the decision to use APL as the CAI 
program language for EQUIP. 

The Objectives of EQUIP 

The second and most Important consideration relating to the 
creation of EQUIP was to determine a sound training foundation and 
minimum Instructional objectives. This process relied heavily on 
the writer's two years of experience while maintaining the QUERY 
system and processing QUERY searches for hundreds of users. Thus 
the central focus used In deriving minimal performance objectives 
for EQUIP are centered on the criteria; What Is the minimum a 
novice user must learn for the creation of a satisfactory computer 
search? Based on experiences with the system, the following guide- 
lines were used: 

(1) The user must understand the proper use and meaning of 
three standard reference sources— the ERIC Thesaurus of Descriptors , 
the ERIC Rotated Thesaurus of Descriptors , and the ERIC Posting of 
Descriptor Statistics (a source of the number of times a particular 
descriptor Is used by RIE). 

(2) The user must be able to create a statement of his 
Information need and subsequently reduce that statement to basic 
elemental descriptors. 
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(3) The user must be able to properly use descriptors to 
Isolate the citations of interest from the ERIC Information files. 

(4) The user must understand and be able to correctly 
use the basic Boolean logic required In the QUERY search language. 

(5) The user must be able to correctly encode a search In 
machine- readable language. 

These objectives were determined to be the minimal learning objectives 
for EQUIP. Since It was desired that the user- to-sys tern Interface 
program be suitable to Individual differences (via CAI)» other less 
Important objectives were evident: 

(1) The user must learn how to use the telecommunication 
terminal . 

(2) The user must be aware of what the computer (QUERY) can 
and cannot do. 

The Development of EQUIP 

Based on the previously defined criteria, EQUIP was planned 
and written. Each subprogram was written and tested using both 
knowledgeable and novice subjects until the specific objectives of 
the subprogram could be met to the satisfaction of the writer 
100 per cent of the time. As would be Imagined, some subprograms 
were rewritten several times before they were deemed acceptable. 

After subprograms and related support programs were written, 
tested, and accepted, the total package was assembled and tested 
using knowledgeable and novice users of ERIC/QUERY facilities. 
Necessary modifications were Installed and "polished" until EQUIP 
was considered ready for formal testing and comparison with existing 
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search negotiations procedures. 

An Integral part of EQUIP Is an eighteen-item pre- and post- 
test administered to the user In the course of his CAI training and 
subsequent search submission. (These measures are discussed more 
fully In Chapter 4.) The prime purpose for the pretest was that of 
switching (branching) the training sequence of the user with EQUIP, 
based on his knowledge of the ERIC system. It was assumed from the 
beginning that the skills and knowledge of users would vary widely. 
Through the use of the pretest switching provision of EQUIP, the 
training a particular user might undergo was, essentially. 
Individually prescribed based on what "he brings to the training 
session." It should be noted that the only major branching with 
EQUIP takes place as a result of the user's performance on the 
pretest. 

A Generalized Description of EQUIP 

EQUIP^ Is comprised of four major Instructional blocks, each 
consisting of subblocks or units. It should be remembered that, 
based on user's knowledge of ERIC or other criteria (familiarity 
with the APL keyboard, etc.). It Is possible that some users received 
no exposure to some blocks or subblocks of the training program. The 
four major blocks and respective objectives are discussed. 



^No attempt will be made to present a comprehensive 
description of EQUIP here. Interested readers will find a complete 
copy of the APL coded programs (functions) in Appendix A and a 
sample of an EQUIP session In Appendix B. 
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Block one; system familiarization . The major purpose of this 



instructional block is to allow the user to learn the basic use of 
the APL telecommunication medium and some very basic concepts unique 
to computer systems. Internal subblocks and objectives are shown in 
Table 2. 

Block two; QUERY familiarization. The second major 
instructional block concentrates on the standard published materials 
deemed necessary to assist the user in his search construction. 

Table 3 depicts the subblocks and objectives of the second instruc- 
tional block. 

Block three; search strategies. Only after the user has 
received appropriate training or has shown that he is knowledgeable 
of the concepts and materials offered in the first two blocks may 
he embark on block three, the most critical portion of the instruc- 
tional sequence. Block three deals exclusively with the construction 
of the user's particular search strategy of interest. Subblocks 
and respective objectives are given in Table 4. 

Block four; practice and submission. The fourth and final 
block of instruction provides the user unlimited practice in creating 
his actual search, encoding it, and, subsequently, submitting it as 
a job to the computer job stream. Additionally, a post test is 
administered, scored, and recorded at the end of this block. Table 5 
presents the subblocks and instructional objectives of block 4. 
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Table 2 

Subblocks and Objectives of Block 1 of EQUIP 



Subblocks 



Objectives 



terminal Training 



Keyboard Training 



Response Training 



Pretest 



Systems Training 



Computer Input 
Familiarization 
Computer Output 
Familiarization 



The user must demonstrate competency In using the 
IBM 2741 telecommunication terminal. 

The user must demonstrate competency In using 
specific APL keyboard characters as well as 
knowledge of basic keyboard functions such as 
correcting mistakes. 

The user must show competency In responding In 
the correct manner to Instructions, questions, or 
commands used within EQUIP. 

A means of ascertaining the user's knowledge base 
relative to the QUERY automated Information 
retrieval system must be provided. 

The user must demonstrate a basic knowledge of 
simple computer input, output, and processing 
concepts. 

The user Is shown how the general search language 
(encoded) appears as Input to QUERY. 

The user Is Invited to Investigate sample 
listings of citations provided near the terminal. 



Note.— With the exception of the last two subblocks, the user 
must demonstrate competencies prior to being advanced further in the 
session. 
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Table 3 

Subblocks and Objectives of Block 2 of EQUIP 



Subblocks 



Objectives 



File Familiarization 



Rotated Thesaurus 
Familiarization 

Descriptor Statistics 
Usage 



Thesaurus Usage and 
Familiarization 



The user must show an acceptable knowledge 
of the use of existing ERIC files and the 
relationship and usage of descriptors in 
Isolating documents from a file. 

The user must indicate that he has learned 
the use of the ERIC rotated thesaurus and 
Its application to modified descriptors. 
The user must demonstrate knowledge and 
use of the ERIC descriptor reports, a 
resource material which aides In 
predicting the number of possible search 
hits. 

The user must demonstrate the ability to 
use descriptors, as given in the ERIC 
thesaurus, In relation to his Information 
need statement. 



Note.— Due to the wide variability of possible correct user 
responses, progress Is monitored through the use of answers given to 
specific questions which are used to switch (skip over) certain 
Instructional material. Progress is not always performance-based as 
was noted in the first block of instruction. 
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Table 4 

Subblocks and Objectives of Block 3 of EQUIP 



Subblocks 


0bje."!.1ves 


Selecting Descriptors 


Using previously learned concepts, the 
user tentatively selects the descriptors 
he actually intends to use for his search 
strategy. 


Logic Training 


Using preselected descriptors from the 
previous subblock, all possible allowed 
logic sequences must be presented and 
understood by the user. It is at this 
point in the instruction that the user 
must indicate his understanding of the 
Boolean logic used by QUERY. 


Estimating Hits 


The user must be cognizant of the 


Familiarization 


possible list his set of descriptors will 
produce, given all possible Boolean logic 
operator combinations. If the user is not 
satisfied, he must have the option of 
either changing his descriptors or 
selecting a different logic pattern. 
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Table 5 

Subblocks and Objectives of Block 4 of EQUIP 







Subblocks 


Object! ves 



Encoding the Search The user must be given the opportunity to 



be actively involved in the encoding of 
his search. 

Diagnostics Rework and Mechanical encoding errors must be detected 
Submission and feedback given to the user as he learns 

to encode his search. The user must 
demonstrate competency in encoding his 
specific search of interest. The user's 
search request must be rejected by EQUIP 
if it fails the search language criteria 
of QUERY. If the strategy is acceptable, 
the user must be allowed to submit his 
search directly to the computer job stream. 
Posttest A means of ascertaining the user's newly 

acquired knowledge of automated infor- 
mation retrieval via QUERY and EQUIP must 
be accomplished 

Note.— The posttest appearing as the last subblock will be 
deleted from EQUIP after final testing of the package. 
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The Instructional Sequence of EQUIP 

EQUIP utilizes a flexible Instruction sequence based on 
initial switching as a result of user responses to the eighteen 
pretest items provided in block one. A simplified flov/chart of 
the sequence decision system is presented in Figure 1. It should 
be noted that not all switching of the sequence is based on pre- 
test responses. Additional switching is accomplished via binary 
choice questions requiring honest user responses. For example, 
if asked "Do you know how to use the rotated thesaurus?" the user 
must respond "No" to receive needed training. Work space limitations 
prevented total competency-basod testing of the user's knowledge— 
thus the "question and response" technique of controlling instruction 
sequence was necessary at times. 

Discussion 

The cumulative elapsed time of instructional block and 
subblock of EQUIP is recorded on two separate time clocks--real 
time and unlocked keyboard wait time (terminal waiting for user 
responses). If the user elects to undergo retraining, when offered, 
his instructional times are accumulated with those of his first 
experience with the subblock concerned. In this manner, not only 
a record of times was recorded for each user but the blocks and 
subblocks actually used in the training sequence were easily 
Identifiable. 

Separate iteni-by-item records were kept of the user's responses 
to both pre- and posttest sessions. These were recorded binarily— 
zero for an item answered incorrectly and one. if answered correctly. 

ERIC ^ 
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Figure 1 

Major Instructional Sequence of EQUIP 
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Keyboard 

training 



Response 

training 



Keyboard 
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Systems 

training 



Computer 

Input 

training 



Computer 

output 

training 



File training 



Thesaurus • 
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training 



Descriptor 

statistics 

familiarization 
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Figure 1 (Contd.) 
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Logic 
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Hits 
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Encode 

Instructions 
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search 



Edit 
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A master real-time clock recorded the total training session 
elapsed time and another clock was employed to keep a cumulative 
record of CPU time used from sign-on to completion of the session. 

A sample copy of clocked times and other monitored variables 
may be found in the last pages of Appendix B. 
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Chapter 4: Procedures 



Design of the Study 

The rational of this study, as developed in the first two 
chapters, reduces to the following experimental question: After 

a CAI interface package has been created, is there any empirical 
evidence that its use with persons desiring information from the 
ERIC information base is effective? To phrase this question in 
another way: Is EQUIP as good an interface technique as existing 

search negotiation sessions with an information specialist? 

Borg and Gall (1971, pp. 363-401) suggest a number of 
experimental research designs of increasing complexity. An 
ideal design, they suggest, is one which offers the maximal 
control of the experiment by the investigator. A suggested 
"acceptable" design for educational research is diagramed below 
(Borg and Gall, 1971, p. 376). 

R 0^ Og (control group) 

R 0^ Xg Og (experimental group) 

Figure 2. An Ideal Research Design 

Briefly, Figure 2 represents a "classical” experimental 
design where subjects in both the experimental and control groups 
are randomly chosen from the population, randomly assigned (R) to 
the treatment groups, pretested (0-|), administered an experimental 
(X^) or control treatment (X^), and posttested CO 2 ). 

Unfortunately, the nature of this study did not lend itself 
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to this highly desirable design structure. Instead, a design 
which offers less control of the experimental situation was required, 
since it was impossible to predetermine a randomly chosen sample 
of subjects from the population of potential users of ERIC/CRESS 
facilities. The offering of computer searches is, by nature, 
service oriented. The subjects, therefore, either do or do not 

desire services — a decision made on an individual basis. This 

! 

characteristic, coupled with the fact that the search negotiation 
interface technique requires no prior knowledge of the ERIC system, 
led to the use of the following experimental design. 

R O 2 (control treatment) 

R IP^ IP 2 O 2 (experimental treatment) 

Figure 3. The Experimental Research Design 

As shown in Figure 3, subjects are randomly selected from the 
population (R), administered an experimental treatment (X^) or the 
control treatment (X^,), and posttested (O 2 ). An interim pre- and 
posttest (P^ and Pg) was administered to the experimental treatment 
group. This is discussed in the appropriate section of this chapter. 

A hazard encountered with the chosen design, suggests 
Borg and Gall (1971, p. 388) is the inability to attribute any 
observed differences noted by posttest results due to the lack of 
baseline data normally determined with a pretest measurement. For 
this reason, the subjects used were those who stated they had no 
previous experience using any automated information retrieval system. 
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This assumption is not detrimental to the investigation of the 
problem, since in the past two years almost all users of ERIC/CRESS 
computer search facilities have been novices. Thus all subjects, 
it is assumed, have the same level of knowledge of the ERIC/QUERY 
information retrieval system— very little, if any. 

Setting of the Study 

This experiment was conducted at ERIC/CRESS, located one 
hundred yards from the newly constructed building housing the 
College of Education on the NMSU campus. Collection of data began 
April 21, 1972, and continued through July 11, 1972. This time 
span allowed the selection of subjects attending long-term semester 
courses as well as summer courses. 

Subjects assigned to the experimental treatment (EQUIP) were 
furnished a new IBM 2741 telecommunications terminal located in a 
well -lighted, adequately^eated and air-conditioned, acoustically 
damped room. The only possible disturbance a subject might 
experience was the occasional noise produced by an IBM 029 
keypunch machine operated in an adjoining room. Each subject 
occupied the terminal room by himself. Since a complete EQUIP 
session might take over two and one-half hours, each subject was 
allowed to set the time he was to use the terminal . 

Subjects assigned to the control treatment were interviewed 
by an ERIC/CRESS information specialist in equally comfortable 
surroundings. Interviews were scheduled at the convenience of the 
subject. 
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Sumnary of the Experimental Design 

Subjects desiring to use the ERIC/QUERY Information 
retrieval system were divided Into two groups— control and 
experimental. The experimental treatments are graphically portrayed 
in Figure 4. 

Both the experimental and control groups underwent a sequence 
of steps, shown as blocks in Figure 4, leading to the retrieval of 
a specific set of surrogated documents (citations) which were 
intended to meet individual informational needs. As depicted in 
Figure 4, the control group sequence (top activities) differed from 
that of the experimental group (bottom activities). Activities 
common to both groups are shovm In the center of the paradigm. The 
activities which differ, search negotiations versus CAI training 
via EQUIP, were those of pivotal interest to the study. 

Sampling Design 

The study population. The population from which the study 
sample was drawn consisted of that group of people who have a need 
for Information in the educational realm and are researchers, 
practitioners, or in training for the same. Additional population 
characteristics were: (a) male or female adults, (b) multi-ethnic, 

(c) must not have used ERIC/QUERY system previously, (d) must have 
been accommodated by the ERIC/QUERY facility, (e) must have been 
willing to spend the appropriate amount of time necessary to 
accomplish a search in person, and (f) must have been aware of ERIC/ 
QUERY facilities. There is no reason to believe that the population 
described here would be different from persons seeking similar 
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Information In other geographic locations of the nation. 

Sampling technique . Since it was not possible to predefine 
a segment of the treatment population in terms of specific 
individuals, a modification of the systematic sampling method 
suggested by Sax (1968, pp. 140-1) and described in subsequent 
paragraphs was used. 

A total sample size of sixty, thirty in each treatment 
group, was deemed necessary. Prior to conducting the experiment, 
a sequential randomized sign-in sheet was created using the random 
number generator feature of APL. The steps of this procedure were; 

(1) A randomly generated vector of sixty numbers ranging 
from 1 to 60 was created. 

(2) Every even position number within the random sixty- 
number vector was isolated from odd position numbers, thus creating 
two separate random independent vectors of thirty numbers each. 

(3) A coin was tossed to determine which thirty-number 
vector would represent the experimental group. In this manner, 
one vector of numbers was assigned to the control group and the 
second was assigned to the experimental group. 

(4) After "tagging” the numbers in each of the two vectors 
with an S (search negotiation) or T (terminal) respectively, the 
sixty numbers were sorted and merged to create a randomized 
treatment assignment sign-in sheet. 

Persons coming to ERIC/CRESS (strictly on a voluntary basis) 
who expressed their desire to use ERIC/QUERY were required to sign 
in on the sequential sign-in sheet described above, after it was 
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ascertained by ERIC/CRESS staff that they were members of the 
study population. In this manner each subject was assigned to a 
treatment group at random. Only chance determined the treatment 
assigned to him. 

Limits of Inferences to the study population . Although there 
Is no reason to believe that the population described previously 
does not exist elsewhere. It would be hazardous to attempt to 
draw Inferences from any population other than that which has access 
to ERIC/CRESS. The sampling design employed allows inferences to be 
made only of this local set of users. No provisions were made for 
Including persons from other locations of the country In the 
sampling design; therefore, any generalizations reported are 
confined to the characteristics of tht members of the educational 
sector of the local study area. 

Measures Employed 

Because of the nature of the search negotiations process 
(the control treatment) It was Impossible to administer a pre- 
or posttest to members of the control group to detennlne any gain 
of knowledge of the ERIC/QUERY system. Since EQUIP was designed 
to Impart knowledge gain, pre- and posttesting was confined to 
subjects In the experimental group. By this procedure. It was 
possible to determine If EQUIP was a successful CAI tool. 

To determine If any difference In user satisfaction of 
search effort was attributable to the experimental treatment, 
a second needed measure was accomplished at the termination of the 
treatment through the use of a user satisfaction questionnaire, 
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which was created and administered to both treatment groups. 

A description of the control group pre- and posttest will 
be discussed first. Similarly, the user satisfaction instrument 
will be detailed last. 

Pre- and posttesting of EQUIP. Due to the uniqueness of 
EQUIP and the necessity of measuring produced knowledge gains, 
coupled with the pretest switching designed Into the programming 
effort, one of the first considerations was that of creating an 
Instrument which could discriminate between a population possessing 
ERIC/QUERY knowledge and skills and one which did not possess such 
attributes. The steps used to creat^ the pre- and posttests 
(Identical instruments) were: 

(1) A "pool" of over thirty items was created based 
on the author's experience with the ERIC/QUERY system. 

(2) A prototype pretest was created on the terminal 
using APL. 

(3) Volunteers from ERIC/CRESS (members of the 
knowledgeable population) and other volunteers not associ ated wi th 
ERIC/CRESS (unknowledgeable population) were administered the 
prototype. 

(4) Prototype Items which did not discriminate between 
knowledgeable and unknowledgeable subjects were deleted and/or 
replaced with new Items until eighteen Items, each meeting the 
measurement criteria relating to the major design objectives of 
EQUIP, were found to discriminate In the desired manner. 
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(5) The final Instrument was again administered to 
different members of the previous population and each Item was 
found to discriminate; thus concurrent validity was assured. 

It will be noted In the Instructional sequence provided 
In Figure 1 (Chapter 3) that the pretest Is administered to 
members of the experimental treatment group Immediately after 
competency In the use of the terminal has been demonstrated by 
the subjects. Similarly, the posttest, which Is Identical to the 
pretest. Is administered after the subject completes his training 
session on EQUIP. 

Using Kuder-Richardson Formula 20, In the manner suggested 
by Sax (1968, p. 161), the final session of the pre- and posttest 
used with the experimental treatment group had a reliability of .73. 
It should be noted that the EQUIP pre- and posttest Instruments 
are "power” and not "speed" tests. The completed Instrument Is 
presented In Appendix B as part of the session sample. 

User satisfaction Instrument . The experimental design 
employed by this study required that a comparative measure of 
treatment groups be administered. Again, due to the nature of 
the study problem. It was Impossible to administer a pretest 
observation to both groups. If a subject had not been served 
there would be no value to measuring his satisfaction with the 
service, since no service would have been received. Thus there 
was a need for only a post-observational measurement to determine 
if the treatments used fostered any differences in satisfaction 
between treatment populations. 






^ A 



o 

ERIC 



42 

It was rationalized that when a user arrives at ERIC/CRESS, 
he has several expectations related to his specific Informational 
need: 

(1) He desires at least some hits as a result of his effort. 

(2) He wants document citations which are relevant to his 
Information need. 

(3) He expects to expend time In his search effort. 

(4) He expects ERIC/CRESS to assist In his search effort. 

(5) He expects his total Information research effort 
to be interesting, challenging, and useful. 

The extent to which the user's Information retrieval experience 
fostered positive and/or negative reactions within each of the 
criterlal characteristics was measured with the simple Likert-type 
questionnaire provided In Appendix C. The questionnaire, the user 
satisfaction instrument discussed previously, was created and tested 
with a small sample of subjects who had just received the results of 
an ERIC/QUERY computer search. Concurrent and content validity were 
established via unstructured Interviews with these subjects. 

Collecting Additional Information 

After each subject signed In (described previously), he was 
directed to complete a one-page form giving his name, subject major 
(If any), college affiliation, phone number, and academic classifica- 
tion. Additionally, he was requested to complete a statement of 
Informational need and to Indicate his retrieval expectations as 
related to number of hits and relevancy of his search. This 
Information collection form may be found In Appendix C. 







Chapter 5: Analysis of Data 



The prime thesis of this study directly relates to three 
specific areas of Inquiry: (a) Is It possible to construct a viable 

us er-to-informatl on-system Interface In a CAI environment? (b) Will 
uninitiated users of such a CAI Interface be as satisfied with 
Information retrieval results as those undergoing traditional search 
negotiations with an Intermediary Information specialist? and (c) Will 
users expecting only a few highly relevant hits (practitioners) be as 
satisfied with their search effort as those who desire a large number 
of multi-relevant hits (researchers)? 

The analysis of data collected for the study will be described 
In five major sections of this chapter. In the first section a 
detailed description of the sample which underwent the study 
treatments Is presented. The second section delineates comparisons 
between experimental and control groups. In the third section the 
performance of the experimental group Is discussed. Additional 
findings related to the scope of the study are given In the fourth 
section, and a brief summary Is provided In the last section. 
Description of the Sample 

Types of users . Lancaster (1968, p. 59) and the Rand 
Corporation (1971, pp. 20-3) strongly advocate functional differences 
between the requesters of Information from an Information retrieval 
system. Generally, users of these systems are classified as 
researchers and practitioners. The distinction between these groups 
Is the expectation each has of the system. Since the literature did 
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not provide any clear definitions for these terms, it was felt that 
a more succinct description of the sample should be extended In this 
chapter of the report. 

As given In the operational definitions (Chapter 1), a 
researcher Is defined as a user seeking a large number of relevant 
to slightly relevant hits, and a practitioner Is a user seeking a 
small number of highly relevant hits as a result of search effort. 

The two types of users, therefore, are separated only by the number 
of expected or desired hits retrieved and the relevancy of the hits 
retrieved. A problem lies In the meaning of "large" and "small" 
numbers of hits and "highly relevant" to "some relevancy". To place 
more direct meaning on these ambiguities, provisions were made for 

I 

subjects to respond to choices of relevancy and retrieval volume 
(see Appendix C) In the manner described In Chapter 4. After all 
data was collected, a comparison of responses to the type of relevancy 
and number of hits desired was conducted. The results are provided 
In Table 6. It will be noted that the number of subjects Indicating 
their desire for "all document citations . . . regardless of how 
relevant . . ." chose values ranging from 1-10 through 50+ as the 
number of desired hits. The same Is true for subjects Indicating 
they wanted "all . . . highly relevant" citations to be retrieved by 
the system. It may be concluded that the key word "all" Is not 
perceived differently by the study sample responding to either of the 
first two choices on the computer search Input Information form. By 
Inspection of Table 6 there can be seen a clear tendency toward 
expecting a greater number of hits from the ERIC/QUERY system by 
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these subjects, the researchers. A definite shift of agreement 
between the number of hits desired and the type of users Is evident. 



Table 6 

Frequencies of Study Sample Subjects In 
Relation to Retrieval Expectations 



Number 

of 

Hits 

Desired 


Type of User 


Researcher 


Practitioner 


Desires 
All Hits 
Regardless 
of Degree of 
Rel evancy 


Desires 

All 

Highly 

Relevant 

Hits 


Desires 
Only a Few 
Highly 
Relevant 
Hits 


Desires Only 
a Few Hits 
Regardless of 
Degree of 
Relevancy 


1-10 


3 (14.29) 


3 (9.38) 


1 (20.0) 


0 (0) 


11-20 


3 (14.29) 


16 (50.0) 


3 (60.0) 


0 (0) 


21-50 


8 (38.1) 


12 (37.5) 


1 (20.0) 


0 (0) 


50+ 


7 (33.3) 


1 (3.12) 


0 (0) 


0 (0) 


Total 


21 


32 


5 


0 



Note.— Values given In parentheses are percentages based on 
column totals. 



the above trend does not manifest Itself with subjects 
Indicating a desire for only a few highly relevant hits. There Is an 
obvious difference, since no experimental subjects Indicated the 
need for over 50 hits while at the same time expecting a "few" highly 
relevant hits. It Is also Interesting to note that of the 58 sample 
subjects, none Indicated they wanted "only a few" hits regardless of 
relevancy. 
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It Is felt that the foregoing discussion legitimizes the use of the 
terms "researcher" and "practitioner" as previously defined. 
Researchers wanted an "In-depth" search with 
numerous document citations; a practitioner did not. 

User characteristics. For the reasons discussed In Chapter 4 
under sampling technique, until the collection of data was completed 
no specific description of the study sample could be provided. Only 
after the experiment was completed and subsequent descriptive 
analysis applied could the sample be more clearly Identified. There- 
fore, It Is felt, such discussion Is appropriate In this chapter of 
the report. 

As stated In Chapter 4 (procedures), 60 novice subjects 
desiring to use the services of the ERIC/QUERY Information retrieval 
system were assigned to either the control group (search negotiations) 
or the experimental group (CAI training and search submission via 
EQUIP). Of the sample of 60, 58 completed the experiment. Subject 
number 17 left the campus without completing the search evaluation 
form and subject number 47 did not complete the CAI training due to 
time commitments. Both these subjects had been assigned to the 
experimental group, but for the reasons stated, both were deleted 
from the sample, thus reducing the total study sample to 30 control 
and 28 experimental treatment subjects. There was no evidence 
Indicating that the loss of two subjects to the experiment was 
due to negative conotations caused by the experimental treatment. 

There was a total of 31 men and 27 women In the sample; 16 
men and 14 women In the control group, 15 men and 13 women In the 
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vexperlmental group. Table 7 shows these and other categories 
expressed as frequencies. 

Table 7 

Study Sample by Treatments 







Characteristic 




Treatment 








Academi c 




Experi - 


Type 


Expectations 


Sex 


Class 


Control 


mental 




Desiring Retrieval 
of all highly 


Men 


Graduate 

Undergraduate 


5 

1 


9 

1 


0 ) 

sz 

u 

u 


relevant document 
citations 


Women 


Graduate 

Undergraduate 


3 

0 


1 

1 


(Q 

0 ) 

0 ) 

oz 


Desiring retrieval 
of all related 


Men 


Graduate 

Undergraduate 


9 

0 


5 

0 




document 

citations 


Women 


Graduate 

Undergraduate 


6 ■ 
3 


8 

1 




Desiring retrieval 
of a few highly 


Men 


Graduate 

Undergraduate 


0 

1 


0 

0 


Q) 

c 

o 

•r- 


relevant document 
citations 


Women 


Graduate 

Undergraduate 


0 

2 


1 

1 


•r- 

-P 

U 

<0 

D. 


Desiring retrieval 
of a few document 


Men 


Graduate 

Undergraduate 


0 

0 


0 

0 




citations regardless 




Graduate 

Undergraduate 


n 


n 




of relevancy 


Women 


u 

0 


u 

0 


Totals 






30 


28 



When it Is remembered that subjects were assigned to the 
respective treatment groups purely by chance, the quantities appearing 
In Table 8 reveal much about the population characteristics. Since no 
descriptive data of ERIC/QUERY users was found in the literature, it 
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Table 8 

Numbers and Percentage of Various Categories 
of Subjects Within Treatment Groups 



Treatment Groups 



Category Control Experimental Totals 

Ny *= 30 Ny = 28 





N 


% 


N 


% 


N 


% 


Researchers 


27 


90.0 


26 


92.9 


53 


91.4 


Practitioners 


3 


10.0 


2 


7.1 


5 


8.62 


Males 


16 


53.3 


13 


46.4 


29 


50.0 


Females 


14 


46. 


15 


53.6 


29 


50.0 


Graduates 


26 


86. 


24 


85.7 


50 


86.2 


Undergraduates 


4 


13.3 


4 


14.3 


8 


13.8 


Male Researchers 


15 


50.0 


15 


53.6 


30 


51.7 


Male Practitioners 


1 


3.33 


0 


0 


1 


1.72 


Female Researchers 


12 


40.0 


11 


39.3 


23 


39.7 


Female Practitioners 


2 


6.67 


2 


7.1 


4 


6.89 


Male Graduate Researchers 


14 


46.7 


14 


50.0 


28 


48.3 


Male Undergraduate Researchers 


1 


3.33 


1 


3.57 


2 


3.45 


Female Graduate Researchers 


9 


30.0 


9 


32.1 


18 


31.0 


Female Undergraduate 
Researchers 


3 


10.0 


2 


7.14 


5 


8.62 


Male Graduate Practitioners 


1 


3.33 


0 


0 


1 


1.72 


Male Undergraduate 
Practi t1 oners 


0 


0 


0 


0 


0 


0 


Female Graduate Practitioners 


2 


6.67 


1 


3.57 


3 


5.17 


Female Undergraduate 
Practi t1 oners 


0 


0 


1 


3.57 


1 


1.72 



Note.— Percentages are given for the respective treatment 
groups while total percentages are based on the total sample (n = 58). 
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was decided, based on similarity of the results of the two Independent 
random subsamples, to present the full set of categories depicted on 
Table 8 , though only the first two categories are considered In 
depth as part of the analysis presented. 

The difference between category frequencies shown on Table 8 
are minute, with the greatest percentage difference found between 
f^ale and male across treatment groups and the smallest between the 
percentage of female practitioners— a range of .47 to 6.9 percentage 
points. This clearly shows that trie two subsamples are almost 
Identical In characteristics, a fact which supports the sampling 
technique employed for the experimental design. 

Comparison of Experimental and Control Groups 

Analysis of measures of user satisfaction . Four of the five 
experiment hypotheses Involve the use of satisfaction scores. Each 
hypothesis will be restated, followed by appropriate data analysis. 

Comparison of user satisfaction Item means . Treatment group 
scores obtained from the five Likert- type scales of the computer 
retrieval evaluation form were compared using the ^ statistic In the 
manner suggested by Winer (1962, pp. 31-33) for comparing uncorrelated 
means of two groups with unequal n's. To test the assumption of 
homogeneity of variance for each measure, the appropriate F ratios 
were computed as suggested by Winer (1962, pp. 33-6). The critical 
value for the two-tailed ^ statistic when .05 and df = 56 
(df *“ Ng + Njj - 2) Is 1 975 ( 56 ) = 2.04, a value obtained from Winer's 
tables (1962, p. 641). The F ratio critical value Is F 95 ( 28 , 30 ) 

* 1.87 from the Chemical Rubber Company (CRC) Tables (1968, p. 308). 
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As was noted In Table 9, the assumption of equal variance of 
group scores for Item three lacks support. This necessitated 
additional analysis of this single measure. Winer (1962, p. 37) 
suggests that If both group sizes are approximately thirty or greater 
= Njj = 30) the ;t' statistic Is an appropriate test of He: 7i=3T2. 
Accordingly, the computations for testing differences between 
Independent sample means with unequal variances were applied to data 
collected for Item three of the user satisfaction questionnaire. A 
summary of these calculations Is given In Figure 5. 

Figure 5 

Summary Calculations of V Statistic for Item Three 
of the Satisfaction Questionnaire 



Treatments 





Control 


Experimental 


N 


30 


28 


Means 


3.7 


3.11 


Variance 


.286 


.692 



Hq: /C3 



I'obs' 3.188135 
c = .28735 
f « 45.53 



The values c and £ are based on Welch's (Winer; 1962, p. 37) 
derivation and approximation of the Student's Jt distribution through 
a correction to the degrees of freedom which are used to ascertain 
the critical _t value. Thus the critical value of using the 
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Table 9 

Summary of Statistical Computations 
Based on User Satisfaction Scores 



Z.X 

n 



Item 


^c 

N*30 


Xe x^-7e 
N=28 


F Ratio 


t 


Hypothesis 
Tested 
(Hi i/s !'/«■,,) 


Satisfaction 
of number 
of hits 


3.00 


2.89 .017 


•988 _ n 

IS?? - 1-023 


0.413 




Satisfaction 
of overall 
relevancy 
of hits 


3.00 


2.75 .250 


Tsif ° 


1.136 




Satisfaction 
of time 
spent 


3.7 


3.11 


.593 


2.417® 


3.251® 




Satisfaction 3.67 
of clearing- 
house effort 


3.82 -.155 


1.003 


-1.076 




General 
Satis fac- 


3.13 


3.00 


.133 




0.604 





tion 



*The F ratio exceeds the critical value of F gg(28,30) * 1.87, 

Therefore H«:d;*'=^*1s rejected. This necessitated further analysis 
0 ®1 

of the treatment means of this measure since the computed ^ might 
not be usable for Inference purposes due to a homogeneity of variance 
violation. 
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appropriate conversion, becomes ^ g 75(f) = 1975(46) = 2.01 for a 
two-tail test. 

It can be seen that the computed 1 value was outside the 
-975 range. Thus it is concluded that there were 

differences manifested between the treatment populations as related 
to the amount of time spent obtaining search results. The testing 
of hypothesis one relating to treatment mean differences as measured 
by analysis of variance (AOV) using total satisfaction scores is 
provided in the next section. 

Hypotheses One, Four, and Five 

It was shown in the first section of the chapter that due 
to user expectations of an information retrieval system, they may 
be classified as researchers or practitioners. Hypotheses one, 
four, and five, restated below, were tested with the application of 
analysis of variance (AOV) technique to determine if a difference in 
interface experiences were related to treatment effect or user type. 

( 1 ) There are no significant differences between the measured 
user satisfaction with computer search results of CAI-interfaced 
subjects and those undergoing search negotiations with an information 
special ist. 

( 4 ) There are no significant differences in user's satis- 
faction betv/een practitioners and researchers as determined by the 
user satisfaction instrument. 

( 5 ) There are no significant interactions between types of 
users (practitioner or researcher) and treatments (CAI or search 
negotiations) as measured by the user satisfaction instrument. 
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Prior to selecting an appropriate analysis technique, the 
following data table was created: 

Table 10 

Summary Table for Experimental and Control Treatment 
Groups Based on Total Satisfaction Scores Obtained 
on the User Satisfaction Instrument 



Total 

N=58 


Researchers 


Practitioners 


Subjects Desiring 
all citations 
regardless of 
relevancy 


Subjects Desiring 
all highly 
relevant 
cl tations 


Subjects Desiring 
a few highly 
relevant 
cl tati ons 


N Y 


N Y 


N Y 


a^ (N=28) 

Experi- 

mental 

Group 

(EQUIP) 


12 15.16667 


14 15.4285 


2 19.0 


ag (N.=30) 

Control 

Group 


9 16.7778 


18 16.6111 


3 15.00 



The design of the study can be viewed as having a 2 x 3 
factorial set of treatments. It can be seen that the six cells of 
the summary table display unequal and disproportionate subclass 
membership; but in such a way that the analysis proposed by Myers 
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Is defensable. Myers states: 

Disproportionate cell frequencies may be representative 
of treatment population frequencies, particularly when 
the independent variable is an individual characteristic 
such as age, sex, socio-economic status, a personality 
trait, or intelligence. In such cases, the method of 
expected cell frequencies may provide an appropriate 
analysis [1966, p. 190]. 

Since a user's expectations from the information retrieval system 
are deemed internal (such as personality and intelligence) and 
independent, the AOV methodology suggested by Myers (1966, p. 190) 
for disproportional cell frequencies with a 2 x 3 factorial experiment 
was followed exactly. This procedure differs from the more common 
least square solution in that cell frequencies and totals are 
recomputed based on the probability of sampling an experimental unit 
from a population of like experimental units. The results of the 
analysis are provided in Table 11. 
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Table 11 

AOV for Testing Hypotheses One, Four, and Five 





Source 


df 


SS 


MS 


F-value 



Total 


58 


16371 .000 






Mean 


1 


14990.07143 






A effect 


1 


12.096233 


12.096233 


.455 


B effect 


2 


3.669421 


1.8347105 


.069 


AB interaction 


2 


24.90709527 12.453547 


.4689 


Residual 


52 


1380.9287 


26.5563 





Note.— The critical F values are F gg{2,52) = 3.15, F gg(l,52) 
=^4.00 which were not exceeded; thus the nonsignificance of the 
related effects. 

Given the procedure and analysis described. It was concluded 
that there are no differences between the responses of the treatment 
groups (A effect), the types of users as classified by expected hits 
(B effect), or interaction between the two factors. Hypotheses one, 
four, and five may not be rejected. 

Performance of the Experimental Group 

Pre- and oosttest measures. To provide evidence that the CAI 
training and search submission program (EQUIP) did Increase the 
knowledge of the user, hypothesis two, re-stated below, was tested In 
the same manner as hypothesis one. 

(2) There are no significant differences between the subjects' 
pre- and post-measures of knowledge of the information retrieval 
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system administered during the experimental CAI treatment. 

Using the procedure suggested by Winer (1962, pp. 39-43) for 
testing hypothesis between two means with correlated observations 
which assume a linear additive model, calculations resulting in the 
values presented in Figure 6 were accomplished. 

Figure 6 

Summary of Calculations Comparing the Pre- and 
Posttest Interface Knowledge Gains of the 
Experimental Treatment Group 



Hypothesis Tested 


^post 


pre 


^post"^pre 


^bs 


H i yus 


79.786 


34.071 


45.714 


14.09 



Since lgg(28) = 2.05, H©: is rejected and it is 

concluded that the difference of 45.714 points average gain from pre- 
and posttest nieasures could not have occurred by chance atoC= .05- 
hypothesis two is rejected. 

Pre- and posttest scores related to satisfaction score . 
Hypothesis three: There is no significant relationship between 

measured knowledge of the ERIC automated information retrieval system 
acquired as a result of CAI training and user satisfaction as 
measured by the user satisfaction Instrument. To investigate the 
possibility of a simple linear regressive association between the 
pre- and posttest gains made by subjects of the experimental group 
and their respective total scores on the satisfaction questionnaire. 




a test of hypothesis three was accomplished. If a predictive 
relationship could be found to exist, additional support for the 
treatment effect on user satisfaction would be gained. Using the 
APL linear regression program MREG provided by the NMSU computer 
center which assumes the linear model Y_ = 3^ + I., two regression 
analysis procedures were accomplished.^ A summary of these efforts 
Is presented In Figure 7. 



^The procedures for this analysis were those suggested by 
Draper and Smith (1966). 



Figure 




The results presented in Figure 7 agree with the scatter- 
grams of the data. By inspecting the plotted values of pre- 
and posttest gains versus total satisfaction score, Figure 8, it 
is fairly obvious why the applied linear regression model accounts 
for only about 20 per cent of the total variation— very little if 
any pattern exists. 

The previous discussion does not explain the significant 
regression manifested by the critical F value of the first 
calculation presented in Figure 7. By careful inspection of Figure 8 
(scattergram of gains versus total satisfaction scores), two isolated 
clusters appear to exist, one in the top portion of the plot and a 
second in the bottom portion. It was felt that further investigation 
was warranted since the relationship of the total satisfaction scores 
to retrieval precision for the subjects isolated in the lower cluster 
of Figure 8 were highly correlated (See Figure 9). 

It is interesting to note that all of the subjects contained 
in cluster 2 obtain a search precision of zero with the exception of 
subject number 18 who obtained a precision of 100. There is 
evidence that this subject's satisfaction was low because, even 
though he obtained maximal precision on his search, he received only 
nine hits but was expecting over 50. From this, we conclude that 
the cluster of subjects providing a lower total satisfaction score 
were those who became disenchanted with the experimental treabnent 
because they received less than expected results from their informa- 
tion retrieval effort, thus influencing the linear regression 
analysis shown to be significant in Figure 7. 



TOTAL SATISFACTION SCORE 
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Figure 8 

Scattergram of Experimental Group Total Satisfaction 
Scores Versus Pre- and Posttest Gains 
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Figure 9 

A Contrast and Correlation of Experimental Total Satisfaction 
Scores and Search Precision for Cluster Two 





Subject 

Number 


Total Satisfaction 
Score 


Precision of Subjects' 
Search Effort 


15 


10 


0 


18 


11 


100 


22 


7 


0 


23 


11 


0 


33 


11 


0 


59 


10 


0 


60 


13 


0 



Correlation = r = .7095 

Basically the question was asked Was the effect of this small 
cluster of subjects obtaining low precision for their search efforts 
responsible for a significant negative slope of the line of the 
regression model? To investigate this possibility further* the two 
groups of subjects In question were separated and the linear analysis 
technique previously described was applied to each to determine If 
any regression effects existed for the isolated cluster. The results 
of these computations are provided In Figure 10. 
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Figure 10 

A Contrast of Linear Regression Sunmary Calculations for the 
Cluster 1 and Cluster 2 Subjects Shown on Figure 8 





Cluster 


Dependent 

Variable 


Independent 

Variable 


»0 

(Intercept) 


(Slope) 


F-value 

(Regression) 


1 

(Top) 

N=21 


Total 

Satisfaction 

Score 


Gain 

Score 


15.8401 


0.364 


1.75801 


2 Total 

(Bottom) Satisfaction 
N=7 Score 


Gain 


9.636 


0.0124 


.0460 



Critical F value for cluster 1 = F gg(l,21) = 4.30 

Critical F value for cluster 2 = F 95(1*7) = 5,59 

Supported by the analysis, it is concluded that the existence 
of a second different sample group within the experimental sample 
(cluster 2, Figure. 8) Is the cause of a significant regression effect 
atcc= .05 when the entire experimental sample's dependent total 
satisfaction scores are predicted using a linear model. However, 
because the R value of 19.7 Indicates that the model accounts for 
only approximately 20 per cent of the variability of the total sample 
responses, and because two clusters, both lacking any significant 
regression effect were shown to exist, hypothesis three may not be 
rejected. 

Additional Findings 

In addition to the observational data applied to each of the 
experimental hypotheses previously discussed,, careful records of 
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the retrieval precision of each subject were maintained. The 
precision of a search effort Is expressed as a ratio converted to 
a percentage of the number of retrieved hits to the total number 
of citations retrieved by the ERIC/QUERY system. For example, a 
precision value of 50 is interpreted as "50 per cent of the citations 
retrieved were considered by the user to be relevant to his given 
information need." Both Lancaster (1968) and Caruso (1969) refer 
to the precision ratio as one measure of an information retrieval 
system desired traits; i.e., the better the ratio the greater 
utility of the system. 

Considerable experience with the ERIC/QUERY system by the 
author has lead to the conclusion that the precision of the ERIC/ 
QUERY system Is less than ideal. Part of the lack of efficacy is 
attributed to the massive size of the thesaurus of descriptors used 
with the system. There are over 4,000 ERIC descriptors which, when 
modified via the TEXT function of ERIC/QUERY may be expanded almost 
to Infinity. 

It was felt that analysis of user's precision values was 
worthwhile. More specifically, does the retrieval precision of an 
expert (the Information specialist) differ from that of a novice (the 
experimental treatment sample)? More formally, this question may be 
stated as a statistical hypothesis: There are no differences 
between the average precision rates of users undergoing CAI training 
and submission via EQUIP and those obtaining the expert services of 
an Information specialist. 



To test this hypothesis, Student's ^ was applied, giving the 
results presented below. 

Figure 11 

Summary of Statistical Computation for Determini ning the 
Significance of Experimental Group Precision Means; 
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Note.— The ratio does not exceed F gg(28,30) = 1.87 and 
does not exceed t 975 ( 56 ) = 2.04. 



The methodology used to obtain the values presented In 
Figure 11 was Identical to that described for testing hypothesis one. 
Based on the non-significant value of it Is concluded that 
H*: based on observed precision values may not be rejected. 
It makes no difference whether a search Is encoded by an expert or a 
novice trained on a CAI Interface program; ERIC/QUERY retrieval 
precision Is uniformly low* — about 55 per cent. 

Summary 

A complete description of the experimental sample was 
presented. Each of five major experimental hypotheses were tested. 
Of the five, only number two was rejected. A sixth hypothesis 
relating to search precision was also tested. Additionally, the use 
of the terms researchers and practitioners as applied to users was 
legitimized. 



Chapters of this report will present an overall summary of 
the experiment, conclusions, recommendations for further study, and 
the specific implications of the statistical tests already presented. 
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Chapter 6: Surmary, Conclusions, and Recommendations 



SunTnarv 

The prime objectives of the study, reduced to more elemental 
partitions, were as follows: 

(1) To develop, Implement, and test a fully documented 
CAI package for training potential users of ERIC/QUERY to Interface 
with the existing system. 

(2) To test the effectiveness of the CAI program produced 
(EQUIP) with the use of pre- and posttest measures. 

(3) To provide an Instrument for measuring user satisfaction 
of his search effort. 

(4) To determine If Interface methods (search negotiations— 
CAI training and search submission) cause a significant difference 

In user satisfaction of experimental and control groups. 

(5) To determine If any Interaction betv/een treatment 
and user-type (researcher and practitioner) factors exist. 

Each of the objectives was accomplished resulting In the 
Interpretations offered In the second section and the conclusions 
discussed In the third section of this chapter. The fourth section 
explores the possible Implication of study results and the last 
section delineates the author's recommendations. 

Interpretations of Experimental Results 

Among treatments . To answer the pivotal Issue of whether or 
not a specially constructed CAI Interface program Is viable, 
statistical tests were conducted to determine If population 
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differences measured by the users' satisfaction responses existed. 
Each of these tests was described in Chapter 4 as related to 
hypotheses one, four, and five. 

Statistical results of the testing procedure clearly indicated 
that the two interface methodologies did not cause any measurable 
differences in user's total satisfaction with the results of his 
search efforts. Analysis of individual items of the satisfaction 
questionnaire showed that, of the five, only one relating to the 
aunount of time spent obtaining a search measured any differences 
between the treatment populations. The experimental group was 
slightly less satisfied with the approximately two and one-half hours 
of time spent using the terminal as compared with control group 
members which spent less than thirty minutes in the search negotiation 
interview. 

It is felt that the measurable difference in satisfaction of 
the amount of time spent by subjects of both treatment groups is non- 
significant in a practical sense. It will be noted in Table 9 on 
page 50 that both mean satisfaction scores for item three, x^ = 3.7 
and x^ « 3.11, are above the mid-point (2.5) of the Likert-type scale, 
suggesting that nei ther experiment group was dissatisfied with the 
time spent obtaining a search. 

Additionally, experimental group subjects returning to use 
EQUIP for a second or third time (on their own and apart from the 
study) were observed initiating searches via a non-training version 
of EQUIP in less than 15 minutes. It is assumed that these subjects 
gained enough skill as a result of their initial usage of EQUIP to 
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accomplish a viable search. This phenomenon has strong implications 
for future research relating to the time a user will spend obtaining 
a search on EQUIP after he has been initially trained. 

It should be noted that EQUIP, as used for this study, consumes 
almost 25 minutes of the user's time (after he has submitted his 
search) for posttest purposes. Unless used for a controlled study, 
the posttest should not be necessary in future versions of EQUIP. 

No interaction between types of users, researchers 
or practitioners, was found to exist. This shows that both types 
of users were equally satisfied with their search efforts and 
results regardless of interface methodology (treatment). 

Within the experimental treatment . A test of the third 
hypothesis relating to pre- and posttest gains of CAI interface users 
indicates that EQUIP does cause a significant increase in the novice 
user's ERIC/QUERY computer-searching skills. Surprisingly, however, 
there is no predictive relationship between such gains and the 
precision of the resulting searches when tested through linear 
regression techniques (hypothesis three. Chapter 4). Such a lack of 
predictability, it was felt, was due to the fact that precision 
attributes of ERIC/QUERY were such that, regardless of the computer- 
searching skill needed to execute a viable search strategy, precision 
rates remained less than ideal. To test this possibility, a separate 
analysis, reported in the latter part of Chapter 5, was conducted to 
determine if the search precision of assumed experts was equal to 
that of novice users trained on the telecommunication terminal. 
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Interestingly, no difference between mean search precision of the 
two groups could be detected. This phenomenon may be due to any of 
three possibilities: (a) the novice users were as skilled as the 

Information specialist; (b) regardless of skill, ERIC/QUERY Is 
uniformly Inefficient; or (c) a combination of both. Unfortunately, 
no empirical evidence may be brought forth which rectifies this Issue; 
however, the results of the precision test between groups supports 
the use of EQUIP. 

In addition to conducting linear regression analysis to 
determine possible predictive relationships betv/een EQUIP gain 
scores and user satisfaction, separate regression analysis was 
conducted for two cluster groups shown to exist within the sample— 
those subjects with very low retrieval rates (cluster 2) and all 
others (cluster 1). Due to the existence of the second cluster. It 
Is evident that total satisfaction scores, and thus the significance 
of total subsample regression line. Is effected, since separate 
regression analysis for each cluster was determined to be non- 
significant. This clearly shows the effect of some users' resulting 
search precision on their overall satisfaction of search effort— 
an understandable situation since users do want access to Information. 
Conclusions 

Based on the procedures and analyses of collected data within 
the scope of the study, the following specific conclusions were 
formulated. These generalizations are confined to the specific 
population used In the study. 



The CAI interface . It is not only possible to use a CAI- 



medium teleprocessing interface with which to bring novice users 
into direct contact with a complex computerized information retrieval 
system, but such an interface method (EQUIP) was created and demon- 
strated to function just as^ well as the search negotiations method 
in use at many installations. It was also demonstrated that 
populations of users classified as researchers or practitioners, 
based on pre-search retrieval expectations of the system, are both 
accommodated with the CAI interface in terms of their satisfaction 
with literature-searching efforts. Thus, the CAI interface services 
both types of users equally well. 

The effectiveness of EQUIP . Although EQUIP is one of many 
possible approaches to training users to submit a computer 
literature search, the significant pre- and posttest means coupled 
with the fact that all of the subjects in the experimental group 
were able to submit a search which was not rejected by the system 
clearly demonstrates the viability of such an interface method. It 
is concluded that EQUIP, as an interface process, did meet the 
objective for which it was created. 

User characteristics . Through the environment of a controlled 
experiment it is evident that at least two types of users, researchers 
and practitioners as defined in this study, may be expected to take 
advantage of automated information retrieval capabilities. It is 
also concluded that the retrieval expectations of these two classes 
of users are different. 
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General . Tests of each of four main stucly hypotheses designed 
to examine relationship between the experimental and treatment 
population resulted In the Inability to reject the null hypotheses; 
l.e.t no differences were found between the experiment groups within 
the dimensions studied. It Is concluded that the CAI Interface 
method, when used In conjunction with ERIC/QUERY, was no more 
nor less effectiva than search negotiations interface even though 
knowledge gains were shown to exist among EQUIP users. 

Implications . 

The use of EQUIP . For the past two years the user desiring 
ERIC/QUERY retrieval services had as his only alternative for 
Initiating a search, the services of an Information specialist— a 
situation which Is manifested In most computerized Information 
retrieval systems desiring to continue receiving the advantages of 
batch processing. With the application of a CAI training and search 
submission Interface, new alternatives will be created. The time 
utilized by an Information specialist to Initiate searches for 
users may confidently be relegated to the student and the computer. 

The number of users accommodated may be Increased In direct 
proportion to the number of accessible telecommunication terminals 
available; thus a larger number of Information seekers may gain 
access to Information retrieval files at a time when the Information 
need Is critical. 

It should be noted that the use of EQUIP, or similar CAI 
training and submission programs. Is not limited to the ERIC/QUERY 
system. EQUIP, with the necessary modifications, may be converted 
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for use with any batch processing information retrieval system 
if minimum hardware and software requirements are provided-- 
specifically, similar APL capability used for the study. Whether 
or not a modified EQUIP or similar CAI interface will be effective 
with other information retrieval systems, of course, is a matter for 
further study. 

Utilization of user characteristics . The detailed sample 
description provided in the first section of Chapter 5 should be 
of some value to future planning related to ERIC/CRESS services. 
Additionally, the application of known user requirements coupled with 
the characteristics provided as a result of the study should be of 
substantial assistance in forecasting user requirements of computer 
facilities. Finally, user reaction to CAI training, as shown by 
their satisfaction scores, were positive— an encouragement to 
explore other CAI/APL applications at New Mexico State University. 
Recommendations 

Before the results of the study may be generalized for other 
areas of the nation, it is recommended that other institutions 
possessing the required capabilities replicate the study. As was 
initially stated in Chapter 1, there is no reason to believe that 
user population in other geographic areas differs from those 

from which the sample was drawn; however, the possibility of 

} 

detectable population differences does exist. 

From the view of one not unfamiliar with APL programming, 
EQUIP ( Version I) as used in the study should be re-encoded. 

Version I consumes approximately twice the necessary storage 



requirements. It Is recommended that each function (subprogram) 
be "compacted" via the use of more efficient program logic and 
APL capabilities. The posttest portion of EQUIP, which consumes 
almost 20,000 bytes of storage and about 25 minutes of user's time, 
should be deleted if further testing of the system Is not anticipated. 

Absolutely no provisions were made for contrasting cost 
differences between the two treatments of the experiment — a highly 
recommended priority for future study. 

Finally, It is strongly recommended that EQUIP or similar 
interface packages be made available to all members of the education 
sector on an "active" basis in contrast to the traditional "passive" 
approach observed in most library settings. Literally millions of 
dollars have been spent creating large, complex, automated 
information systems. Through the use of a viable CAI interface, 
the decision-maker, researcher, practitioner, student, or teacher 
would have no one to blame for not accessing needed information but 
himself. 
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COMPUTER SEARCH INPUT INFORMATION 



NAME-. : ,• . CLASSIFICATION: 

{LAST) {FIRST) U IR D SR 

GRAD ; □ M 

MAJOR: . □ 5P □ DR. 

COLLEGE: . UCC CHARGE CODE: 



DATE: I 172 PHONE: . SEX: UM DP 

USING 1 TO 2 SENTENCES, GIVE YOUR SPECIFIC INFORMATIONAL NEED: 






INDICATE YOUR SEARCH PREFERENCE {CHECK QiE. 3: 

□ WANT DOCUMENT CITATIONS TO BE RETRIEVED BY THE SYSTEM 
REGARDLESS OF HOU REl'S.VANZ {I.E. HIGHLY RELEVANT AS NELL 
AS SLIGHTLY RELEVANT) SUCH CITATIONS MIGHT BE TO THE INFOR- 
MATIONAL NEED STATED PREVIOUSLY . 

n NANT iLL OF THE DOCUMENT CITATIONS VHICH ARE UlQUl 
MliKlillZ TO YOUR STATED INFORMATIONAL NEED. 

□ WANT ONLY A {I -E. A SAMPLE) OF THE DOCUMENT CITATIONS 

WHICH ARE IILQIIL'I REL^VArjS. TO YOUR STATED INFORMATIONAL 
NEED . 

□ WANT ONLY A lEW DOCUMENT CITATIONS REGARDLESS OF HOW 
RELEVANT TO YOUR STATED INFORMATIONAL NEED THEY MAY BE. 

glZS: A '^HIT^' IS ANY DOCUMENT CITATION RETRIEVED BY THE COMPUT- 
ER WHICH IS RELEVANT TO YOUR INFORMATIONAL NEED. 

HOW MANY "HITS^^ DO YOU FEEL YOU MIGHT NEED TO SATISFY YOUR 

INFORMATIONAL NEED? {CHECK QMl 

□ 1 - 10 □ 11 - 20 □ 21 - 50 □ 50 0/? MORE 



CLEARINGHOUSE USE ONLY: 




CRBE: U BIB Dl D2 D3 0* 



D C 



□ E NO 



COMPUTER RETRIEVAL EVALUATION 
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NAME : 



DATE-. I '72 



PLEASE INDICATE TOUR SATISFACTION WITH THE FOLLOWING BY CIRCLING- 
QNE. OF THE NUMERIC CHOICES GIVEN ON EACH SCALE. 

A 'UlIT^' IS ANY DOCUMENT CITATION RETRIEVED BY THE COMPUTER 
WHICH YOU DEEM RELEVANT TO IQUR INFORMATIONAL NEED. 

HOW DO YOU FEEL ABOUT THE EILI1S.ER OF HITS RETRIEVED BY YOUR 
SEARCH? 



4 

A 

1 


2 

1 


3 

- 1 


4 

1 


VERY 

DISSATISFIED 


DISSATISFIED 


SATISFIED 


VERY 

SA TISFIED 


HOW DO YOU FEEL . 


ABOUT THE 


RRLRIllQl OF 


THE HITS RETRIEVED 


BY YOUR SEARCH? 

1 

1 

VERY 

DISSATISFIED 


2 

1 


3 

1 


4 

1 


1 

DISSATISFIED 


SATISFIED 


VERY 

SATISFIED 



HOW DO YOU FEEL ABOUT THE AMOUNT OF ZUIR ^OU SPENT {I.E. FROM THE 
TIME YOU ENTERED THE CLEARINGHOUSE UNTIL YOUR SEARCH STRATEGY WAS 
READY FOR SUBMISSION TO THE COMPUTER)? 



1 

1 

VERY 

DISSATISFIED 


2 

1 


3 

1 


4 

1 


DISSATISFIED 


SATISFIED 


VERY 

SATISFIED 


HOW DO YOU FEEL ABOUT THE AMOUNT OF 
EXTENDED TOWARDS YOUR SEARCH NEED? 


REIQRZ the 


CLEARINGHOUSE 


1 

1 

VERY 

DISSATISFIED 


2 

1 


3 

1 


4 

1 


DISSATISFIED 


SATISFIED 


VERY 

SATISFIED 



iPLEASE TURN TO NEXT PAGE .1 



iC\3L 



f 
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IN GENERAL, HOW DO 
SEARCH EFFORT? 


YOU FEEL ABOUT 


THE RESULTS 


OF YOUR COMPUTER 

ff 


1 

1 


2 

1 


3 

1 


4 

- 1 
VERY 

SATISFIED 


1 

VERY 

DISSATISFIED 


DISSATISFIED 


_ 1 

SATISFIED 



HOW !UHY HITS IVOX NECESSARILY TOTAL RETRIEVED CITATIONS'] DID YOU 
RECEIVE? 

HOW MNY CITATIONS DID YOU RECEIVE WHICH WERE HQZ RELEVANT TO 
YOUR INFORMATIONAL NEED? 

HOW MANY TOTAL CITATIONS {.HITS "JUNK'n DID YOU RECEIVE? 



IF, AT ANOTHER TIME, YOU HAVE OTHER INFORMATIONAL NEEDS WHICH 
MAY BE SERVED BY ERIC, WILL YOU RETURN FOR ANOTHER COMPUTER 
SEARCH? D YES □ NO 

IF NO, WHY? ' ^ 



WHAT IMPROVEMENTS WOULD YOU LIKE TO SEE MADE BY THE CLEARING- 
HOUSE TO MAKE COMPUTER SEARCHING OF ERIC FILES BETTER? 



IF YOU WERE ADVISING A FRIEND CONCERNING THE COMPUTER SEARCHING 
SERVICES OFFERED AT THIS CLEARINGHOUSE •. 

WHAT WOULD BE THE STRONGEST POINT YOU WOULD DISCUSS? 



WHAT WOULD BE THE WEAKEST POINT? 



CLEARINGHOUSE USE ONLY'. 

□ CDF NO, TOT □ FP □ VAR 






/ 
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QUERY Soft/are 

The specific batch processing retrieval system used for the 
study was that provided by Computer Resources Corporation (CRC), 

6825 Redmond Drive, McLean, Virginia, as modified by ERIC/CRESS. The 
CRC version of QUERY was F08APR70. The search parameters defined 
as minimum and used by both experimental groups were: (a) a maximum 

of five descriptors allowed, (b) no single descriptor searches 
allowed, (c) only the "or" and "and" search logic operators were 
allowed, (d) standard QUERY input formats were required, and (e) 
the TXT parameter was used throughout. 

EDP Environment 

Hardware . The following computer hardware, part of which was 
utilized during the study, was available at the NMSU Computer Center. 

* IBM 360/65, 256K 

* 2M bytes Ampex ECM (1.4 microsec interleaved) 

* 2 Ampex disk spindles (30 microsec average access) 

* 3 selector channels 

1 Mpx channel w/2 selector subchannels 

* 1 1403 printer 

The items marked with an asterisk (*) are critical capabilities but 
were not necessarily dedicated for the study. Additionally, it is 
known that the follovnng reduced EDP envi ronment wi 1 1 accommodate 
EQUIP and ERIC/QUERY system; however, substantially less response 
time may be expected. 

IBM 360/50, 256K 

IM bytes IBM LCS (8 microsec) 
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2 No. 2314 disk spindles (60 microsec average access) 

2 selector channels 
1 Mpx channel 
1 1403 printer 

System software . The NMSU computer utilizes IBM's OS version 
NVT, Release 19.6 in conjunction with HASP Version 3.0. The APL 
capability is IBM's Version 1, level 0 as modified by Dr. Tom H. 
Puckett. The APL workspace size is 35,000 bytes each. EQUIP 
consumes seven such v/orkspaces for a total 245,000 bytes of which 
210,000 are maintained on on-line storage devices (disk) at any given 
time. Dedicated active workspace within the Ampex ECM numbers to 
three before "swap out" occurs. The APL supervisor resides in 
IBM 360/65 core. 

System work load . The NMSU multiprogramming system 
accommodates an average of 1,000 jobs per deyr— a situation which 
was not found to be detrimental to the study application. 




133 



210 



ERIC DAT A ELEMENTS AHD THEIR FIELD IDENTIFICATION NUMBERS 
RESUME SUBSYSTEM 



ORDERED BY FIELD IDENTIFICATION NUMBER 



DATA ELEMENT/FIELD 


KEYWORD 


USED IN* 


HEXADECIHAt 


DECIMAL 


NOTES 


Sequence 


None 


F J P R(ALL) 


00 


0 




Add Date (Ju 1 ian) 


None 


F J P R(ALL) 


01 


1 




Change Date (Julian) 


None 


F PR 


02 


2 




-.VAccession Number 


ACC 


F J P R(ALL) 


‘ 10 


16 




■••‘Clearinghouse Acc, No. 


CH 


J R 


n 


17 




Other Accession Number 


OCH 


R 


12 


18 




Publication Type 


PUBTYPE 


R 


13 


19 


OBSOLETE 


Program Area 


PA 


P R 


14 


20 




Proposal Date 


PROPDAT 


P 


15 


21 




Project Date 


PROJDAT 


P 


16 


22 




Publication Date 


PDAT 


R 


17 


23 




Total Project Dollars 


TOT 


P 


18 


24 




Fiscal Year Funding 


FYF 


P 


19 


25 




Title 


TITL 


J P R 


lA 


26 




Personal Author 


AUTH 


J R 


IB 


27 




Inst i tut ion 


INST 


F PR 


ic 


28 


CODE 


Re s pons i b 1 e B ranch 


BRANCH 


P 


ID 


29 


CODE 


Geographic Locality 


GEO 


F P 


lE 


30 


CODE 


Cooperating Institution 


CINST 


P 


IF 


31 


CODE 


Sponsoring Agency 


SPON 


P R 


20 


32 


CODE a 


Fund i ng Agency 


FUNDG 


P 


21 


33 


CODE ^ 


Funded Individual 


FUNDED 


P 


22 


34 




Descriptors 


DESC 


F J P RfALL) 


23 


35 




Idem i f lers 


IDEN 


F J P R(ALLJ 


24 


36 




EDRS Price 


PRICE 


P R 


25 


37 


- 


Descriptive Note 


NOTE 


R 


26 


38 




Project Officer 


PRO 


P 


27 


39 




Principal Investigator 


PRI 


P 


28 


40 




Field Reade r 


READ 


F 


29 


41 




Address 


ADDR 


F 


2A 


42 




1 s sue 


ISS 


F J P R(ALL) 


2B 


43 




Abstract 


A8ST 


F J P R(ALL) 


2C 


44 




Report Number 


REPNO 


R 


20 


45 




Contract Nuriber 


CONT 


F PR 


2E 


46 




Grant Number 


GR 


P R 


2F 


47 




Bureau Number 


BN 


P R 


30 


48 




Avai labi 1 1 ty 


AVAIL 


J R 


31 


4q 




Journal C i tat i on 


JML 


J R 


32 


50 




EKperience Highlights 


EH2 0.EIII 


F 


33-46 


51-70 




Pub 1 i cat i ons 


PUBL 


F 


47 


51 




Education 


EDUC 


F 


48 


72 




Region 


REG 


F 


49 


73 




Project Nuniber 


PN 


P 


5A 


74 




Project Category 


PCAT 


P 


5B 


75 




Institution Name 


None 


F PR 


80 


128 


28 


Responsible Br. Name 


None 


P 


81 


129 


29 


Geographic Loc . Name 


None 


r p 


82 


130 


( Oerived 30 


Cooperating Inst. Name 


None 


p 


83 


131 


( F rom 3 1 


Sponsoring Agency Name 


Ncne 


P R 


84 


132 


{ Field 32 


Funding Agency Name 


None 


P 
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33 


Region Name 


None 


F 


AO 


173 


73 



f = Personnel; J * Journal Articles; P » Projects; R “ Reports 





9 



vj 



ERIC MASTER DATA F ILES MAGNETIC TAPE F ORMAT 21 1 

( 80 ) ( 80 ) 



OS/360 

TAPE 

LABEL 



Vol ume 


Header 


Header 


Tape 


1 

DATA 


Tape 


Trai ler 


Trai 1 er 


Tape 


Label 


Labe I 
1 


Label 

2 


Mark 


Mark 


Label 


Label 


Mark 



























\ 






\ 



\ 



\ 



\ 



\ 



BLOCK 



Block 
Byte ' 
Count 







Variabl e 


Variable 


Variabl e jj 






Length 


Length 


Length // etc. 






Record 1 


Record 2 


Record 3 H 



Reserved 
for O.S.^' 

/ 

/ 

/ 

/ 



\ 



\ 



\ 



\ 



\ 



\ 



\ 



RECORD 



Field I 



Record 

Byte 

Count 



7 \ Reserved / 
for O.S,/ 



Field 2 



etc. y/ Field N 



FIELD 




Fieldy i ti Field Identification 
Byte' Number 



Count 



in parentheses indicate the length in bytes of fixed length data, 



O 
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\ 



"Sir 



* Numbers 



